Today's networks have substantially increased the usefulness of computers such as personal computers (PC's). A local-area network (LAN) such as an Ethernet can connect several PC's together, and an Internet router or gateway on the Ethernet can allow all PC's to reach the larger Internet.
Peripherals and other devices such as keyboards and mice may be connected to an individual PC using smaller, less complex networks such as Peripheral Component Interconnect Express (PCIE) or Universal-Serial-Bus (USB). Such peripheral devices were traditionally daisy-chained together, but more recently USB hubs or PCIE switches are being used. These are often known as load-store buses since they operate within a single address domain, and write and read data or packets to and from memory addresses often using input and output CPU instructions or memory read and memory write CPU instructions.
FIG. 1 shows a PC connecting to peripherals using a PCIE switch. PCIE switch 20 has several ports that connect to PCIE links 22. Host processor 10, such as a processor on a PC, acts as the host or root controller for all PCIE transactions. PCIE media-access controller (MAC) 12 converts commands and data from host processor 10 to PCIE packets that are transmitted serially over PCIE link 22 to PCIE switch 20.
Similarly, each peripheral such as video 18, disk 14, and Ethernet NIC 16 has its own PCIE MAC 12 for communicating over a PCIE link 22 to PCIE switch 20. Video 18, disk 14, and Ethernet NIC 16 act as PCIE slave devices while host processor 10 acts as the master. Each PCIE link 22 has two pairs of signal wires that carry full-duplex differential data in both directions. Power and ground and a clock may also be present in cables for PCIE links 22.
PCIE switch 20 switches data among ports for PCIE links 22 as required.
While PCIE is useful, the PCIE standard allows for only one host with a single address space, or a single address domain. Different host computers typically each have their own address space or domain, and may use addresses within that domain without regard to use of these addresses in other address domains. Connecting two or more hosts together using PCIE is problematic and may required specialized hardware and software. Communication buffers such as I/O ports may be used between address domains. An extension of PCIE known as Advanced Switching (AS) may also be used.
FIG. 2 shows a blade server using an Ethernet switch to connect blades. A blade server typically has several computers located together on a common rack or chassis. An Ethernet switch may be located in one of the chassis slots in the blade server. Ethernet links are made between each blade computer and the Ethernet switch, allowing Ethernet communication among the blade's computers. An Ethernet network interface controller (NIC) may be located on the same board as a blade's processor and memory.
For example, blade A has processor 10 and Ethernet NIC 32 on first blade board 34. An economical blade system can be made by using a modified PC motherboard for each blade computer. First blade board 34 operates as a separate PC system, with processor 10 acting as the host processor. Communication with other blades and the outside world is primarily made using Ethernet NIC 32, and input, output, or peripheral devices connected to first blade board 34 are minimal or non-existent, or used only for diagnostics.
First blade board 34 connects to Ethernet switch 30 through Ethernet physical link 36, which can be a twisted-pair cable plugged into receptacles or jacks on first blade board 34 and on Ethernet switch 30, such as RJ-45 jacks, or an optical fiber, a wireless link or a differential copper trace on a PCB. Ethernet switch 30 contains many ports, while blade boards typically have only 1 port to connect to a first switch, and a second port as a redundant connection. Ethernet switch 30 typically resides on a removable board, a special slot designed for a switch.
Other blades on blade boards 34′, 34″, 34′″ also have processors 10′, 10″, 10′″ and Ethernet NIC's 32′, 32″, 32′″ and connect to Ethernet switch 30 by Ethernet physical links 36′, 36″, 36′″, respectively. Since Ethernet is a widely-used communication standard, inter-processor communication within a blade system is facilitated. Ethernet switch 30 may also have ports that connect to an external network such as a corporate local-area network (LAN) or a bridge to the Internet.
FIG. 3 illustrates in more detail Ethernet hardware in a blade system. Ethernet switch 30 is located in a blade system, such as in one of the chassis slots. Ethernet NIC's are located on each individual blade computer's board in the other chassis slots of the blade system. For example, blade A has Ethernet NIC 32 that connects to Ethernet switch 30 over Ethernet physical link 36, while blade C Ethernet NIC 32″ that connects to Ethernet switch 30 over Ethernet physical link 36″.
Each blade's host processor may communicate with it's Ethernet NIC over a local expansion bus such as PCI, PCIE, USB, or some other load-store bus. Host interface 38 receives commands and data from the host processor over this local bus, which may be stored in registers such as Ethernet control registers. Ethernet data may be located in a main memory used by the host processor, and these registers may contain pointers to the data that host interface 38 uses to read the data when an Ethernet packet is being assembled and transmitted.
The assembled data is stored by host interface 38 into blade packet FIFO 42, which is read by Ethernet media-access controller (MAC) 44 for transmission over blade Ethernet physical layer 46. Ethernet MAC 44 performs media-access layer functions such as checksum generation and either collision detection with re-transmission, or full-duplex throttle-back functions. Other functions may also be performed, such as comparing destination addresses of received packets and discarding packets mis-matching the controller's MAC destination address. Ethernet physical layer 46 contains the physical line drivers that drive serial data onto Ethernet physical link 36.
On Ethernet switch 30, switch Ethernet physical layer 48 contains circuits such as line receivers and clock re-generators to receive analog data and convert it to digital data. Switch Ethernet MAC 52 verifies checksums and removes low-level headers, writing the packet data into switch packet FIFO 54.
Switch logic 56 then determines the destination MAC address from the received Ethernet packet and requests a link be made through cross-bar 40 by switch controller 50. A routing table may be consulted by switch controller 50 to determine which switch port receives packets for that Ethernet MAC address. Switch controller 50 then instructs or configures cross-bar 40 to establish a connection from the input port to the output port on Ethernet switch 30. The Ethernet packet is then copied from the input port's switch packet FIFO 54 to the output port's switch packet FIFO 54′.
The Ethernet packet is then sent out the output port by Switch Ethernet MAC 52′ reading from switch packet FIFO 54′ and sending the Ethernet packet to another blade computer's Ethernet NIC 32′ over another Ethernet physical link 36′ using switch Ethernet physical layer 48′. The receiving blade B's Ethernet physical layer 46′ receives the data from Ethernet physical link 36′, sends the data up to Ethernet MAC 44′ which writes it to blade packet FIFO 42′. Finally host interface 38′ sends the data from the Ethernet packet to the blade processor or its memory. Thus blade A sends an Ethernet packet to blade B through Ethernet switch 30.
A shared memory may be used for cross-bar 40, or a hardware cross-bar switch may be used. Other implementations may also be used in various Ethernet switch architectures. When a shared memory is used, switch packet FIFO's 54, 54′, 54″ may be parts of the shared memory.
While blade systems with Ethernet switches are quite useful and widely used today, there is an inherent inefficiency in using Ethernet for communications among blade computers in a rack. Data to be exchanged among blade computers is stored in their local memories, converted to PCIE frames and copied to blade packet FIFO 42 over a local PCIE bus, then passes down through Ethernet MAC 44 and blade Ethernet physical layer 46, across Ethernet physical link 36 and back up through switch Ethernet physical layer 48 and Switch Ethernet MAC 52 before being switched through cross-bar 40.
Then the packet data goes back down through the switches' link and physical layers of Switch Ethernet MAC 52′ and switch Ethernet physical layer 48′, over Ethernet physical link 36′, then back up blade Ethernet physical layer 46′ and Ethernet MAC 44′. Then the data is carried over PCIE frames on the PCIE bus of blade B to the memory of blade B's processor.
Also, the Ethernet MAC's are often specific to a network speed, such as 10/100 for 10 mbps and 100 mbps, of 1 G for 1 Giga-bit-per-second, etc. As network speeds increase, upgrades to the MAC's are necessary.
What is desired is a more efficient Ethernet switch for blade systems. An Ethernet switch is desired that switches using Ethernet MAC addresses, yet connects to host computers in a more efficient manner.